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ABSTRACT 



The purpose of this study was to examine doctoral students' 
preparation in statistics in the field of education. A national survey was 
conducted focusing on 27 quantitative methods (QM) programs. One QM professor 
from each program was identified and asked to describe and evaluate the 
training of QM and non-QM doctoral students at his or her institution. The 
vast majority of professors indicated that most or all of the students in 
their QM programs received training in the "old standard" procedures, 
analysis of variance, multiple regression , and traditional multivariate 
procedures, while fewer than half of the professors indicated that most or 
all of their QM students received training in more recent procedures such as 
bootstrapping and multilevel models. Professors were also asked to rate the 
skills of the QM students' training in areas such as mathematical statistics 
and computer skills on a scale from "Weak" to "Strong. " Most professors gave 
high marks to their QM students' skills with statistical packages, but gave 
much more mixed ratings of their QM students ' training in mathematical 
statistics. Nearly half of the professors thought that most of their QM 
students could have benefited from one to two additional statistics courses. 
Results are discussed in terms of training of future doctoral students. 
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Abstract 



Although numerous research studies have focused on issues related to the teaching of 
statistics, few studies have focused on the training of people who may become statistics teachers. 
The purpose of this study was to examine doctoral students' preparation in statistics in the field 
of education. A national survey was conducted of twenty-seven quantitative methods (QM) 
programs. One QM professor fi-om each program was identified and asked to describe and 
evaluate the training of QM and non-QM doctoral students at his or her institution. The vast 
majority of professors indicated that most or all of the students in their QM programs received 
training in the "old standard" procedures— ANOV A, multiple regression, and traditional 
multivariate procedures, whereas less than half of the professors indicated that most or all of 
their QM students received training in more recent procedures such as bootstrapping and 
multilevel models. Professors were also asked to rate the skills of their QM students' training in 
areas such as mathematical statistics and computer skills on a scale fi-om "Weak" to "Strong". 
Most professors gave high ratings to their QM students' skills with statistical packages, but gave 
much more mixed ratings of their QM students' training in mathematical statistics. Nearly half of 
the professors thought that most of their QM students could have benefited fi-om 1 to 2 
additional statistics courses. Results are discussed in terms of training fijture doctoral students. 
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Training Graduate Students in Educational Statistics: 

A National Survey 

There are several indications that the level of methodological and statistical 
sophistication required of graduates of doctoral programs in education has been increasing 
over the years. In terms of educational statistics, we have witnessed the relatively recent 
introduction of a wide variety of advanced statistical methods into the field of education, 
including meta-analysis, bootstrapping, multilevel modeling, and structural equation 
modeling. We have also seen the field of qualitative methods develop from a small 
educational research camp to a major educational research paradigm. It is no longer 
unusual to find an issue of a research journal where one article utilizes complex statistical 
procedures and a second article utilizes a rigorous qualitative analysis. For example, 
journals as diverse as Exceptional Children, TESOL Quarterly (Teaching English to 
Speakers of Other Languages), md Journal for Research in Mathematics Education 
consistently publish both quantitative and qualitative research. A well-rounded graduate 
of a doctoral program in education thus needs a remarkable breadth and depth of 
knowledge of research methods and statistics, whether it be to critically evaluate research 
articles, to conduct research, to write and evaluate grant proposals, to serve as editorial 
board members for scholarly journals, or to supervising master's theses and doctoral 
dissertations. As faculty members who are responsible for training doctoral students in 
research methods and statistics, a key question then becomes how well are we actually 
preparing these students for their future roles as researchers and research consumers. 

This paper focuses on one aspect of doctoral student preparation, namely 
preparation in educational statistics. ^ In particular, the purpose of this current research is 
to conduct a survey of doctoral students' training in statistics. This survey addressed the 
statistical training of two groups of doctoral students, namely those students who 
specialize in quantitative research methods (loosely defined to include doctoral students 
interested in research design, educational measurement, and educational statistics), and 
those students who specialize in fields other than quantitative research methods. 
Examining the training of this first group, namely doctoral students specializing in 
quantitative methods (QM), was considered important because these students may 
become tomorrow's leaders in quantitative methods, taking on responsibilities such as 
training future doctoral students in statistics, measurement, and research methods; 
publishing original research in statistics, measurement, and research methods; providing 



^Ideally, we would have also considered doctoral students' training in measnrement and research methods 
(including qualitative methods). We chose to focus on one aspect of training, namely educational statistics, 
to make this research project more manageable in scope. 
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methodological advice on master's thesis and doctoral dissertation committees; serving as 
methodological consultants to educational journals; and so forth. Examining the training of 
this second group, namely non-QM doctoral students, was also considered important, 
because these doctoral students may eventually become leaders in substantative areas of 
educational research. 

Background 

There is limited literature on doctoral students' training in educational statistics. A 
survey of statistics course content for doctoral students has never been conducted in the 
field of education, although studies of statistical preparation have been conducted in 
psychology (see Akin, West, Sechrest, and Reno, 1990) and medicine (see Dawson- 
Saunders, Azen, Greenberg, and Reed, 1987). The lack of systematic research on what 
doctoral students are learning in their statistics classes in the United States seems 
particularly unfortunate, because this information could help facilitate discussions of 
whether this training should be restructured and, if so, how it should be restructured. This 
is not to say that discussions of what should be taught do not take place. For example, 
Tukey (1980), among others, has argued that students need much more experience with 
exploratory data analysis in their statistics training. As a second example, Noether (1980) 
has argued for the importance of teaching nonparametrics in introductory statistics classes, 
arguing that it can provide a strong conceptual framework for students. Brogan (1986) 
similarly suggested that nonparametrics should be taught, especially in social science and 
nursing disciplines. Brogan further recommended that in a required two-course statistics 
sequence, graduate students should learn about general linear models, including multiple 
linear regression, ANOVA, and possibly categorical data analysis, and should also be 
trained to use computer statistical packages. 

Content analyses of the types of statistics and research designs currently being 
used in published research can serve as a foundation for discussions of what we should be 
teaching. A number of such content analyses have been performed over the years. Elmore 
and Woehlke (1988), for example, conducted a content analysis of statistical methods used 
in the American Educational Research Journal (AERJ), Educational Researcher, and 
Review of Educational Research for all articles published between the years of 1978 to 
1987. For AERJ, they found that the predominant statistical method used in these articles 
was ANOVA/ANCOVA, which was used in 137 out of 396 articles that they coded. The 
next most fi-equent procedure was multiple correlation/regression, which was used in 24% 
of the articles; followed by multivariate procedures (13%), bivariate correlation (12%), t- 
tests (11%) nonparametrics (12%), and structural equation modeling (10%). Assuming 
that the statistical analyses found AERJ are typical of those that doctoral students in 
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education encounter, then it would seem reasonable to ask about the extent to which we 
are training doctoral students to conduct and interpret research which utilizes these 
procedures. 

Aiken, West, Sechrest, and Reno (1990) examined the adequacy of doctoral student 
preparation in statistics in their own field, namely psychology. Because our research is 
similar to theirs, we will describe it in detail. In particular, Aiken, West, Sechrest, and 
Reno surveyed all 222 psychology departments or schools identified by the American 
Psychological Association as granting doctoral degrees. From these 222 units, 186 
responded, for a response rate of 84%. The goals of their research were (a) to describe the 
current content of the statistics, measurement, and research design courses; (b) to look at 
differences in requirements between sub-disciplines in psychology; and (c) to gather data 
on professors' perceptions of students' abilities to apply statistical and measurement 
techniques in their own research. In addition, these researchers distinguished between 
"elite" and "other" institutions, and presented their data comparing the two types of 
institutions. 

In terms of demographics, Aiken, West, Sechrest, and Reno (1990) found that 89% 
of the departments offered introductory graduate statistics sequence, and of the 
departments offering this sequence, 77% were one year in length. A minority of 
institutions— 17%— offered doctorates in a quantitative area of specialization, and one-third 
of these 17% did not have any first-year students. In terms of curriculum offerings 
(including optional and required courses), the researchers found that in statistics courses, 
the majority of programs offered at least a partial course (defined as at least a half a 
semester or fall quarter) in ANOVA (88% of the programs), multiple regression, (58%) 
and MANOVA (53%). More recent statistical topics such as structural equation models 
and time series models were covered in a minority of programs (18% and 6%, 
respectively). In terms of the content of the required introductory statistics course 
sequence, the authors distinguished between "old standards of statistics" and "more 
advanced statistical considerations", and collected information on the percent of 
institutions offering in-depth coverage of specific procedures. ("In-depth" was defined as 
coverage to the point that students could perform the analysis in question themselves). 
In-depth coverage of the "old standards" was provided in the majority of institutions: 
multifactor ANOVA, multiple comparison procedures, repeated measures via traditional 
factorial ANOVA, and multiple regression were covered in-depth in the required course 
sequence in 73%, 69%, 73% and 63% of the institutions, respectively. Topics considered 
more advanced tended to be covered in-depth in required courses much less often. For 
example, 39% of the institutions provided in-depth coverage of ANCOVA in their 
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required course sequence, 38% provided in-depth coverage of ANOVA as a special case of 
regression; 20% provided in-depth coverage of exploratory data analysis, 21% provided 
in-depth coverage of multivariate procedures, and 18% provided in-depth coverage of 
statistical power analysis. The researchers interpreted these findings by stating that 

the statistical and methodological curriculum has advanced little in 20 years; 
measurement has experienced a substantial decline. Typical first-year courses serve 
well only those students who undertake traditional laboratory research... New PhDs 
are judged to be competent to handle traditional techniques, but not newer and often 
more useful procedures, in their own research, (p. 721)". 

Our purpose was similar to those of Aiken, West, Sechrest, and Reno (1990), 
namely to conduct a survey of doctoral students' training in statistics. First, we were 
interested in learning about the kinds of statistical methods and procedures in which QM 
students and non-QM students in education are trained. Second, we were interested in 
learning about QM professors' perceptions of QM and non-QM doctoral students' 
competence with various statistical techniques. 

Methods 

Sampling Procedure and Instrumentation 

The universities we surveyed are prominent educational research institutions, 
namely those institutions identified in a University of Illinois study as the top thirty-one 
Colleges of Education ranked in terms of academic productivity and prestige (West & 

Rhee, 1994). A list of these institutions is provided in Table 1. Our goal was to identify 
one QM professor from each institution as a contact person who would complete the 
questionnaire. If a program had a QM program, we selected a faculty member affiliated 
with this program. In order to identify these contacts, several strategies were used. First, 
we simply identified people that we knew personally at specific universities, with the 
belief that these people would be most likely to complete the questionnaires. Second, we 
used the membership roster of the AERA Educational Statisticians Special Interest Giroup. 
Because the results of this study would be presented at the AERA conference, we believed 
that using this membership roster would increase our chance of having the questionnaires 
completed and returned. Third, we used a book entitled "Glraduate Study in Educational 
and Psychological Measurement, Quantitative Psychology, and Related Fields", published 
by the Pennsylvania State University, which lists quantitatively-oriented programs in the 
United States, and provides names of appropriate contact persons. Finally, in a few 
instances, we used personal contacts to identify a non-quantitative faculty member at a 
particular university who then gave us the name of a person who taught statistics at that 
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university. There were only two instances where we could not identify a contact at a given 
university, leaving an effective population of 29 universities. 

Our intention was to collect data from all 29 universities, regardless of whether or 
not they had quantitative methods (QM) programs. Moreover, we decided to be flexible 
in terms our definition of what a QM program is— thus, for example, some programs might 
place a heavy emphasis on measurement, whereas others might place a heavy emphasis on 
educational statistics. To figure out which universities had a QM program, we also used 
our personal knowledge of various universities. In cases where we were not sure if a 
university had a QM program, we attempted to contact someone at that university. Out 
of the 29 universities, all but 2 had QM programs. (These two Schools/Colleges without 
QM programs still offered doctoral training in statistics.) 

Two versions of the questionnaire were developed. The first version was a four- 
page questionnaire, and was designed for the 27 universities that had a QM program. For 
these 27 universities, the QM faculty were asked to evaluate both the QM and non-QM 
doctoral students. The second version of the questionnaire was a one-page shortened 
version of the full questionnaire, and was designed for the 2 universities which did not 
have a QM program. For these 2 universities, the QM faculty were asked to evaluate the 
non-QM doctoral students only. A copy of the full-length questionnaire is provided in 
Appendix 1. 

In November of 1995, all participants received a copy of the questionnaire (which 
will be described later) to complete. A second mailing was conducted approximately one 
month later, in December of 1995. In January 1996, this second mailing was then followed 
by e-mail and/or phone calls asking that participants return their questionnaires. All 
questionnaires received within 3 months of the initial mailing were then analyzed. In terms 
of the number of responses, 20 of the 27 universities with QM programs responded, and 1 
of the 2 universities without QM programs responded. In total, 21 out of 29 universities 
responded, for a response rate of 72%. We believe that this response rate is large enough 
to generate credible results, although it is possible that there is some unknown but 
systematic difference between QM faculty members who responded to the survey and 
those who did not respond. 

Designing the questionnaire: QM students. For the portion of the questionnaire 
addressing the training of the QM doctoral students, the process of questionnaire 
development was relatively straight-forward. For this group, we collected the following 
data; 
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(a) demographic information, including information on the number of full-time 
faculty in the QM program, the number of lecturers in the QM program, the 
number of students in the QM program, and the where on the campus students 
took their required statistics courses (e.g. in the School/College of Education, 
Statistics department, etc.), 

(b) information on the extent to which QM doctoral students have been trained in 
specific statistical topics or methods, 

(c) information on professors' opinions about the adequacy of QM doctoral 
students' computer skills and mathematical training, and 

(d) information on professors' opinions on whether the QM doctoral students could 
have benefited from more statistics coursework. 

Designing the questionnaire: Non-QM students. For the portion of the 
questiormaire addressing the training of the non-QM students, the development of the 
questiormaire was more challenging because the questiormaire items needed to be 
sufficiently flexible to accommodate the variety of ways in which Schools and Colleges of 
Education are organized. Specifically, a College of Education might consist of a series of 
schools, divisions, departments, or other organizational units, each with separate 
programs. Each organizational unit could conceivably have its own statistics requirement, 
with its own statistics course sequence. A non-QM students with one area of 
specialization at a particular institution could therefore take one required statistics course 
taught in one particular organizational unit, whereas another student with a different area 
of specialization could take a different required statistics course taught in a different 
organizational unit. 

Considering all of these potential complexities, it became apparent to us that 
complete survey of what non-QM students learn in their statistics courses would require 
that ^ of the statistics instructors be surveyed in each organizational unit at each 
institution. We thought that such a survey would be too unwieldy, and decided to shift 
the focus for the non-QM students away from a survey of course content and course 
requirements and towards a survey of quantitative professors' perceptions of non-QM 
doctoral students' preparation in educational statistics. We asked one professor at each 
institution to evaluate doctoral students' training in educational statistics at his or her 
institution and to frame this evaluation in terms of doctoral students with whom he or she 
had direct contact with over the past few years— through committees, research 
assistantships, classes, and so forth. In this way, we hoped to circumvent potential 
problems of (a) having professors speculate about what was taught in statistics classes 
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they might not have thorough knowledge about, and (b) having professors speculate about 
the statistical competence of doctoral students whom they might not know. 

For the non-QM students, we collected the following data: 

(a) information on statistics requirements, 

(b) information on whether statistics requirements have changed over the past 5 
years, 

(c) information on professors' perceptions of doctoral students' ability to critically 
read and interpret research articles utilizing specific statistical procedures, and 

(d) information on professors' opinions on whether the non-QM doctoral students 
could have benefited from more statistics coursework. 

Limitations of the Current Research 

Before presenting the results, we believe that it is important to point out three 
main limitations of the current research. First, the questionnaire was not extensively field 
tested, and no reliability or validity information was obtained for the questionnaire items. 
In fact, several of the items had to be excluded from the analysis because of apparent 
problems. Second, only one QM faculty member was surveyed from each institution, 
thus limiting the generalizability of the results. Third, because of time and resource 
constraints, we did not collect survey data on doctoral students' preparation in the areas of 
educational research methods or measurement, thus limiting the picture we can paint about 
doctoral student training. But, on the positive side, we believe that the survey has several 
advantages, among them the facts that (a) it is not particularly time consuming research 
method, and (b) it is not particularly intrusive. 

Results 

I. Statistics Preparation of QM Students in Education 

Demographics. First consider the data for the QM programs and students. 
Demographic information is presented in Table 2. In terms of faculty size, the median size 
the full-time tenure and tenure-track faculty in these QM programs is 5, with 4 full-time 
faculty regularly teaching statistics during the academic year. In terms of student 
enrollment in QM programs, the distribution was positively skewed, but typically, 
approximately 4.5 students were admitted annually during the previous 5 years. In terms 
of the location of the required statistics courses, all of the institutions required students to 
take at least one statistics course in the School or College of Education; 50% required 
students to take one or more required courses in the Mathematics or Statistics 
Departments; and a small number (20%) required students to take one or more required 
courses in the Psychology Department. 
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QM students' preparation in specific statistical topics and methods. We were 
interested in learning about the extent to which doctoral students in QM programs received 
training in specific topics and procedures. We developed a list of procedures which we 
thought reflected a breadth of statistical procedures, and also included procedures we 
considered to be more "recent", such as meta-analysis, bootstrapping/jackknifing, 
multilevel models, and causal models. We then asked faculty to mark those procedures 
and methods in which most or all of their QM students received training. Results are 
presented in Table 3. 

The data suggest that the majority of QM doctoral students receive training in the 
"old standards"— ANOVA, multiple regression, and traditional multivariate procedure. 

The exceptions to this appear to be logistic regression and log-linear models, where about 
half of the respondents indicated that their QM students receive training. Nonparametric 
procedures, on the other hand, appear to be less prevalent. Although a more detailed list 
of nonparametric procedures was not provided, it seems reasonable to use the Kruskall- 
Wallis test as a benchmark, and to note that only half of the respondents indicated that 
their QM students received training in this procedure. Finally, in terms of more recent 
procedures, we see that the majority of programs train their students in meta-analysis 
(70%), and over one-third of the programs train their students in causal models (50%), 
jackknifing/bootstrapping (40%), and multilevel models (35%). 

Mathematical statistics training and computer skills of QM students. We 
also asked the faculty to rank the mathematical statistics training and computer skills of 
most of their QM graduates. We used a 4-point scale with "1" indicating "Weak" skills 
and "4" indicating "Strong" skills. Results are presented in Table 4. In terms of the 
mathematical statistics training, faculty ratings were symmetrically distributed, with most 
of faculty rating their students midway on the scale, either as a '2' (44% of faculty) or as a 
'3' (44% of faculty). In terms of computer skills, the vast majority (90%) thought their 
students had "strong" skills in standard data analysis programs such as SAS and SPSS. 
Faculty's perceptions of student competence in other computer skills— database 
management and programming— were much more varied. 

Comparing the mathematical statistical training of past and present QM 
graduates. Colleagues have occasionally remarked that they believe that earlier graduates 
of QM programs were better trained in terms than more recent graduates. We were 
interested in investigating these kinds of comments, by asking faculty if they thought that 
doctoral students who had graduated in the last year or two were better prepared or less- 
well prepared in terms of statistics than students who graduated 5 to 6 years earlier. More 
specifically, we were interested in knowing if they thought there was a difference over 
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time in QM graduates' abilities to do publishable statistical work. Data from Table 5 does 
not support the claim that earlier graduates were perceived to be stronger. In fact, none of 
the faculty indicated that thought that their recent QM students were less-well prepared 
than graduates from 5 to 6 years prior. Over half of the faculty (60%) thought that there 
was no difference between past and present graduates, and the remaining 40% believed 
that recent graduates are better-prepared. 

Faculty members' perceptions of QM students' needs for additional statistics 
and qualitative methods coursework. Finally, we asked faculty members to indicate 
whether they thought their students could have benefited from additional (probably 
specialized) statistics coursework and from additional qualitative coursework. Data are 
presented in Table 6. In terms of statistics, nearly half of the respondents (44%) indicated 
that they thought that over half of their students could have benefited from 1 to 2 
additional courses; whereas 22% indicated that they thought that over half of their 
students could have benefited from 3 or more additional courses. Few of the respondents 
(11%) thought that their students could have benefited from additional qualitative 
coursework. 

n. Statistics Preparation of non-QM Doctoral Students in Education 

Statistics requirements. For the non-QM doctoral students, our first interest was 
to see if each institution had a minimal statistics requirement for all doctoral students in 
education. Descriptive statistics are presented in Table 7. In nearly half of the 
institutions (43%), students in some programs can graduate without taking a statistics 
course whereas in 40% of the institutions, students in all programs are required to take at 
least one statistics course^,. In only 14% of the institutions is there a uniform 2-course 
statistics requirement. 

We had speculated that students might be taking fewer statistics courses currently, 
in comparison to previous years, due to the increased prevalence of qualitative research 
methods. Data in Table 8, however, does not indicate a decreasing trend in statistics 
enrollment over the past five years. In about one-third of the institutions, students are 
taking fewer statistics classes, but in an equal number of institutions, they are taking more 
classes. Finally, data from Table 7 indicate that in only 14% of the institutions is there a 
uniform qualitative methods requirement. ^ 



^Note that this statistic does not provide an estimate of how many students actually graduate without a 
statistics course in a given institution. 

^Note that this statistic does not provide an estimate of how many students actually take a qualitative 
methods course. 
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Perceived competence with specific statistical procedures. We also asked faculty . 
to rate non-QM doctoral students in terms of their ability to critically read and interpret 
research articles utilizing specific statistical procedures (see Table 9). In particular, faculty 
were asked to think about the non-QM doctoral students they had contact with over the 
past five years, and to indicate whether they thought more than half or less than half of 
these students would be competent with a particular statistical method. Notice that this 
question requires a lower threshold of "competence" than a question that might ask about 
doctoral students' abilities to carry out and analyze research utilizing these procedures. 

The procedures listed— including ANOVA, OLS regression, MANOVA, and 
nonparametric rank tests— were not meant to be an exhaustive list of procedures, but rather 
were selected because we thought they represented a cross-section of "common" and 
"advanced" statistical methods found in published educational research. 

Results indicate that the faculty perceived the students to be most competent with 
ANOVA: 68% of the faculty thought that more than half of their graduates could 
critically read an article using ANOVA. Only 38% of the faculty thought that more than 
half of their graduates could critically read an article using OLS regression. Faculty from 
only 1 or 2 of the 21 institutions thought that more than half of their students could 
critically read and interpret articles that utilized more advanced (and less common) 
procedure, namely MANOVA, log-linear models, nonparametric rank tests, and causal 
models. 

Faculty members' perceptions of graduates’ need for additional statistical and 
qualitative coursework. Faculty members were asked to judge the extent to which they 
thought that non-QM students who had graduated from their School/College of Education 
could have benefited from additional statistics coursework. More specifically, faculty 
were asked to think about the students they had contact with over the past 5 years who 
had graduated, and to gauge whether more than half or less than half of them would have 
benefited from additional statistics coursework. Results are shown in Table 10. Virtually 
all of the faculty thought that more than half of the graduates could have benefited from 
additional (probably specialized) statistics coursework: 43% of the faculty thought that 
over half of the graduates could have benefited from 1-2 additional courses, and 48% of the 
faculty though that at least half of the graduates could have benefited from 3 or more 
courses. In terms of qualitative methods, the majority of faculty (62%) had no opinion on 
whether graduates could have benefited from more coursework, and only 2 faculty thought 
that at least half of the graduates could have benefited. 

Discussion 
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Has this survey accomplished anything usefiil? Despite the methodological 
shortcomings discussed previously, we so. Consider first the research findings related to 
the preparation of doctoral students in QM programs. 

Implications of the research on QM doctoral students. For the portion of the 
survey addressing the preparation of QM students, we offer three main implications for 
our findings. First, we now have empirical evidence of the nature of the curriculum 
offered in a sample of QM programs. We found much of this information encouraging 
because it indicated widespread professional agreement in the way that doctoral students 
majoring in quantitative methods are trained. For example, almost all of the surveyed 
programs emphasized traditional data-analytic procedures, trained students to use 
computer packages, and thought that recent graduates were as well or better trained than 
earlier graduates. 

Another finding of interest was that a number of QM programs offered instruction 
in more recent or advanced procedures such as meta-analysis, causal models, and 
jackknifing/bootstrapping. This implies that a number of faculty believe that this kind of 
training is important enough to incorporate into the curriculum. Another implication of 
our findings is that their similarity to those of Aiken, West, Sechrest and Reno (1990) 
suggests that information fi'om the two studies can be combined to draw stronger 
conclusions and guide fijture research. 

Perhaps the most important implication of our research on QM programs is that 
the findings offer a platform from which we can ask (and in some cases provide tentative 
answers to) some important questions in the training of doctoral students in quantitative 
methods. For example, the answer to the question "What percentage of quantitative 
methods programs have incorporated newer techniques into their curriculum within the 
past 10 years?" appears to be about half of those surveyed. In our view, this places our 
findings in a somewhat less positive light. What accounts for the failure to offer training 
in these newer techniques? Is it simply the result of constraints on credit hours, too few 
faculty, or that training students who are not majoring in quantitative methods is now the 
primary business of many quantitative methods programs and that this has a severe effect 
on the curriculum? And is it acceptable that virtually none of the surveyed programs 
required training in qualitative research methods yet prominent research journals in 
education regularly publish the results of qualitative research studies? 

We believe the questions raised above (and others) are important and thus provide 
directions for additional work in the area of preparing doctoral students in quantitative 
methods. Perhaps the most obvious work to be done is to widen the participation of QM 
programs and to obtain information about the kinds of positions taken by graduates. As 
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the statistical education literature has argued, curricula must be responsive to the needs of 
the institutions and industries hiring graduates. We believe that the emergence of 
information about the training of students in quantitative methods programs should 
encourage a dialog among professional educators that is overdue. 

Preparing non-QM doctoral students. Now consider the portion of the survey 
that addresses the preparation of non-QM students. We believe that our research findings 
are useful in that they provide descriptive information on faculty members’ perceptions of 
doctoral student competence. Specifically, the research findings suggest that a number of 
QM faculty think that a good portion of non-QM doctoral students are under-prepared to 
be critical consumers of tj^ical quantitatively-oriented research articles. We found that 
3 1% of the faculty thought that less than half of the non-QM doctoral students could 
critically read and interpret research articles utilizing ANOVA, and 62% of the faculty 
thought that less than half of the non-QM doctoral students could critically read and 
interpret research articles utilizing OLS regression. These two procedures--ANOVA and 
regression— are fairly common in educational research literature. (Recall the work of 
Elmore and Woehlke (1988), who found that in AERJ, the predominant statistical method 
used was ANOVA/ANCOVA, which was used in 35% of the articles coded, followed by 
multiple correlation/regression, which was used in 24% of the articles.) Moreover, the 
vast majority of faculty thought that less than half of the non-QM doctoral students 
could critically read articles utilizing more advanced procedures, namely MANOVA, log- 
linear models, nonparametric rank tests, and causal models. 

These research findings suggest that we may have cause for concern about non- 
QM doctoral students’ preparation in statistics. Educational researchers have a variety of 
job responsibilities, including guiding their students' research; writing and reviewing 
research grant proposals; serving as educational research consultants; serving as editorial 
board members; and conducting research in their area of specialization. If a graduate of a 
doctoral program in education has difficulty critically reading and evaluating quantitative 
journal articles, then it seems possible that this graduate may also have difficulty 
functioning successfully as an educational researcher. 

Although these findings on the preparation of non-QM doctoral students are only 
tentative, they do offer direction for future research and discussion. One line of research 
suggested by this investigation involves studying faculty members' opinions about why 
some non-QM doctoral students at their universities are under-prepared in educational 
statistics and what they think can be done to improve non-QM doctoral students' 
preparation in educational statistics. Another line of research suggested by this current 
investigation involves studying editors of educational research journals. Because these 
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X O 



editors review so many manuscripts (both publishable and unpublishable), they are in a 
good position to evaluate the statistical and methodological training of today’s educational 
researchers. In addition of journal editors, it might prove fruitful to study the opinions of 
the doctoral students themselves regarding the adequacy of their own training. Lastly, the 
perceptions of non-QM faculty could also be of value in informing a discussion on the 
statistical preparation of non-QM doctoral students. 
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Table 1 

Universities Initially Contacted for this Study 

1. University of Wisconsin— Madison 

2. University of Illinois 

3. Ohio State University 

4. Stanford University 

5. University of Minnesota 

6. Indiana University— Bloomington 

7. Michigan State University 

8. Columbia— Teachers College 

9. University of Georgia 

10. Peimsylvania State University 

1 1 . University of Maryland 

12. University of Texas- -Austin 

13 . University of Michigan 

14. Arizona State University 

15. University of California— Los Angeles 

16. University of Washington 

17. University of California— Berkeley 

18. University of Chicago 

19. Harvard University 

20. University of Virginia 

21. Vanderbilt University— Peabody 

22. University of North Carolina 

23. University of Florida 

24. Florida State University 

25. Syracuse University 

26. University of Nebraska 

28. Virginia Pol34echnic 

29. SUNY-Bufifalo 

30. University of Missouri 

3 1 . University of Pittsburgh 



Note: The final sample consisted of N=21 universities from the above list. 
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Table 2: Demographics: Quantitative Methods Programs 



Quantitative Methods Program Size 

•Number of full-time tenured/tenure-track faculty in QM 



program: 




Mean 


5.3 


Median 


5.0 


SD 


2.3 


N 


20 


•Number of full-time faculty who regularly teach statistics 


during the academic year: 




Mean 


3.7 


Median 


4.0 


SD 


1.8 


N 


20 


•Number of lecturers and adjunct faculty members who 


teach statistics in QM program: 




Mean 


1.4 


Median 


1.0 


SD 


2.3 


N 


20 


•Approximate number of doctoral students admitted to 


QM program each year over past 5 years: 




Mean 


7.9 


Median 


4.5 


SD 


9.3 


N 


19 


Location of Required Statistics Courses’*’ 




in OM Prosram fN=20) 


Pet 


School/College of Education 


100% 


Mathematics/Statistics Department 


50% 


Psychology Department 


10% 


Other 


5% 


Academic Year Svstem tN= 20) 


Pet 


Quarter 


20% 


Semester 


80% 



*Note; These percentages add to more that 100 because some programs require courses in 
a School of Education, for example, and a statistics department. 
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Table 3 


Preparation of Quantitative Methods Doctoral Students (N = 20) 


Percent of Universities Where 


Most or All QM Program Graduates 


Receive Training in Topic/Method 


ANOVA 


Covariance analysis 


100% 


Repeated measures designs 


95% 


Mixed-effects models 


95% 


Power/sample size calculations 


95% 


Random-effects models 


85% 


Nonorthogonal designs 


80% 


Thorough coverage of multiple comparison procedures 


75% 


Cell means models 


55% 


Complex designs (e.g. fractional factorial) 


40% 


Multiple Regression 


Ordinary least squares estimation 


100% 


Weighted least squares estimation 


70% 


Nonlinear -in-the-predictors models 


70% 


Logistic regression 


60% 


Nonlinear-in-the-parameters models 


35% 


Traditional Multivariate Procedures 


Canonical correlation 


90% 


MANOVA 


90% 


Discriminant analysis 


90% 


Factor analysis 


85% 


Principal Components Analysis 


80% 


MANCOVA 


80% 


Log-linear models 


50% 


Nonparametric Procedures 


Kruskall-Wallis test 


50% 


Exact tests 


45% 


Jackknifing/Bootstrapping 


40% 


Repeated measures tests (e.g. Friedman, Cochran, 


30% 


Hodges-Lehmann block test) 


Asymptotic relative efficiency 


25% 


Rank-transform tests 


20% 


Other Topics and Procedures 


Matrix algebra 


85% 


Meta-analysis 


70% 


Causal Models 


50% 
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Multilevel models/Hierarchical linear models 
Times series models 



ERIC 



35 % 

5 % 
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Table 4 



Faculty Members Perceptions of the Mathematical/Statistical Training and 
Computer Skills of Quantitative Methods Program Graduates 



Weak Strong 







(1) 


(2) 


( 3 ) 


( 4 ) 


Mean 


SD 


N 


1. 


Mathematical/statistics training of most 


1 


8 


8 


1 


2.5 


.71 


18 




QM graduates.** 


(6%) 


(44%) 


(44%) 


(6%) 








2. 


Computer skills: Standard data analysis 


0 


0 


2 


18 


3.9 


.31 


20 




programs (e.g. SAS, SPSS, SYSTAT) 


(0%) 


(0%) 


(10%) 


(90%) 








3. 


Computer skills: Database management 


1 


6 


12 


1 


2.7 


.65 


20 






(5%) 


(30%) 


(60%) 


(5%) 








4: 


Computer skills: Programming 


6 


4 


9 


1 


2.2 


.95 


20 


(e g. FORTRAN, PASCAL, C) ( 


[30%) 


(20%) 


(45%) 


(5%) 









**For this question, the anchors were as follows: 4=Strong, perhaps took multiple courses in 
mathematical statistics, probability theory, would probably be well acquainted with topic such as 
quadratic forms. Likelihood Ratio Test Principle, Gauss-Markoff Theorem, etc; and 1 = Weak, little or 
no coursework in mathematical statistics or probability theory. 



Table 5 

Faculty Members Perceptions of QM Program Graduates' Ability to do Publishable Statistical 

Work* (N = 20) 



Pet 

•Faculty who believe that recent graduates are 

better prepared than graduates from 5-6 years 40% 

ago. 

•Faculty who believe that recent graduates are 

less-well prepared than graduates from 5-6 years 0% 

ago. 

•Faculty who believe that there is no difference 

between recent graduates and graduates from 5-6 60% 

years ago. 

*Note: "Recent graduates" were defined as people who graduated in the last year or two. 
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Table 6 

Faculty Members' Perceptions of Student Needs for Additional Statistics and Qualitative 
Methods Coursework for Quantitative Methods Program Doctoral Students (N = 20) 



Faculty Members 
Who Believe That More 
Half of Graduates 

Could have Profited Faculty Members 

from Additional Course with No 

(>50%) Opinion 



Statistics Coursework 
•1-2 additional (probably specialized) 
statistics courses (N = 1 8) 

•3 or more additional (probably specialized) 
statistics courses (N = 18) 

Qualitative Coursework 

• 1 to 2 additional qualitative methods courses 
(N- 18) 



Pet 


Pet 


44% 


11% 


22% 


17% 


11% 


17% 



*Note: Data are in response to the following question: "In hindsight, of the quantitative 
methods program doctoral students who graduated from your program in the last 
5 years, what percent do you think could have profited from (the following 
additional coursework)?" 
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Table 7 

Coursework Requirements for Doctoral Students in Education who 
are not in Quantitative Methods Programs 



Faculty 

Selecting Option 



1. Statistics Requirement (N = 21) Pet 

•Students in some programs can graduate without taking a statistics course 43% 

•Students in all programs are required to take at least one statistics course 38% 

•Students in all programs are required to take at least two statistics courses 14% 

•Don't know 5% 

2. Qualitative Research Requirement (N = 21) Pet 

•Students in some programs can graduate without taking a qualitative methods course 76% 
•Students in all programs are required to take at least one qualitative methods course 14% 
•Don't know 10% 



Table 8 

Changes in Statistics Enrollment over the Past Five Years 
for Doctoral Students in Education who are not in Quantitative Methods Programs 

Faculty 

Selecting Option 



Enrollment Change over the Past Five Years (N = 21) Pet 

•Doctoral students have tended to take fewer statistics courses 33% 

•Doctoral students have tended to take more statistics courses 33% 

•Doctoral students have tended to take about the same number of statistics courses 24% 

•Don't know/No Response 10% 
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Table 9 



Perception of Students' Competence to Critically Read and Interpret Research 
Articles Utilizing Specific Statistical Procedures for Doctoral Students who are 
not in Quantitative Methods Programs'^ 

(N = 21) 



Faculty Members Faculty Members 

Indicating that Less than Indicating that More than 

Half of the Graduates are Half of the Graduates are 



Competent Competent 

(<50%) (>50%) 





N 


Pet 


N 


Pet 


•ANOVA 


8 


(31%) 


13 


(62%) 


•OLS regression 


13 


(62%) 


8 


(38%) 


•MANOVA 


19 


(91%) 


1 


(5%) 


•Log-linear models 


20 


(95%) 


1 


(5%) 


•Nonparametric rank tests 19 


(91%) 


2 


(10%) 


•Causal models 


19 


(91%) 


1 


(5%) 



*Note: Data are based on responses to the following questions: "Think about the 
doctoral students in your School/College of Education who are NOT in your 
quantitative methods program that you have had contact with over the past 5 
years or so— through dissertation committees, classes, consulting, and so forth. Of 
these non -quantitative students, what percent do you think are competent to 
critically read and interpret research articles that utilize the following procedures? 
(1) Less than half, (2) More than half, (3) Can't judge. 
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Table 10 

Faculty Members' Perceptions of Graduates’ Needs for Additional Statistics and Qualitative 
Methods Coursework for Doctoral Students who are not in Quantitative Methods Programs (N 

= 21 ) 



Faculty Members 
Who Believe That More 
Half of Graduates 
Could have Profited 



from Additional Coursework 




(>50%) 


Statistics Coursework 

•1-2 additional (probably specialized) 


Pet 


statistics courses (N = 21') 


43% 


•3 or more additional (probably specialized) 
statistics courses (N = 21') 


48% 


Qualitative Coursework** 

•1 to 2 additional Qualitative methods courses 


10% 



(N=21) 



*Note: Data are in response to the following question: "Think about the doctoral 

students in your School/College of Education who are NOT in your quantitative 
methods program that you have had contact with over the past 5 years or so— 
through dissertation committees, classes, consulting and so forth. Of these non- 
quantitative students, what percent do you think could have profited from (the 
following additional coursework)". 

** For this question, 13 faculty members had no opinion. 




26 









V 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Ubraiy of Education (NLE) 

Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 

(Specific Dcwument) 




i. DOCUMENT IDENTiFiCATiON: 



Title; 'Jra>r\xn^ QraJoaic + \or\c,{ 

f] i^a-\\or>£(( Sur\j<y 

Author(s): f)- ConAu t^^clr\c^cl H ar<^e.lj 

Corporate Source; 



S i iT-f >cs : 



PubficaSon Date; 



ii. REPRODUCTION RELEASE: 

!n order to disseminate as widely as possible timely and significant materiats of interest to the educational community, documents 
announced in the monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, 
reproduced paper copy, and electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source 
of each document, and, if reproduction release is granted, one of the following notices is affixed to the document. 

If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following lh.me options and sign 
at the bottom of the page. 



The sarhple sUcxer shown belO’<v will be 
to art Level 1 docurneets 



The san^e sficker shov.'n beiow w:l be 
anrxe<5 :o all Levei 2A (documents 



The samii^e sfioker shov,<n be:ow wi:l be 
afTfxed io all Level 2B <Socuments 




PERMfSSsON TO REPRODUCE AND 
D^SEMiNfATE THIS MATERIAL IN 
MICROFICHE, AND :N ELECTRONIC ?.4EOtA 
FOR ERIC COLLECTION SUBSCRiSERS ONLY, 
HAS BES'vt GRAT^TED BY 

TO THE EDUCAnONAL RESOURCES 
INFORMATION CENTER (ERiC; 



2A 



Level 2A 

Q^eck. here for Le'vel 2A reteese. permitting ro{^ 0 (tuc;tor) 
and dissen%na^or> t>. nntcrctiche end in etecbonic media for 
ERIC archival coL'ectlon subscribers onty 



PER^<JSS:ON TO REPRODUCE AND 
D;SSEMiNATE THIS MIATERIAl IN 
MICROFICHE Of^iLV H.AS SEEN GRANTED BY 



TO THE EDUCAnONAL RESOURCES 
INFORMATION CENTER {’ERIC; 



2B 



Level 2B 

(^)eck h^e for Level 28 release. cermIttEng repmduc^n 
and dissemination rcecroflche only 



C'Ocuments will be pmcassed as indicated pro'^^ded reproctectlon guaiily pemvfs. 
if permissJon to r&prcdyca Is granted, but no box is checked, itoct^nents twJl be processed at Level 1. 



/ neraby grant to the Educaftonaf Resources information Center (ERIC) ncnexc/us/Ve permission (o reproduce and disseminate this document 
as indicated ebove, Repfxyduction hvm the ERiC microfiche or eiecfronic media by persons other than ERiC emptoyees and its system 
contractors requires permission from the copydghf hoider. Hxcepfron is made for norhprofit reproduction by tibraries and other service agencies 
fc? satisfy information needs of educators in response to discrete inquiries. 


§ oscr 


Sisnalure; , ^ jn * 

JiQru/i 


Pfintec Nflmsff>o£itjoni'7i:te: 

’bz.Bcnnk ftssocia-h 


Organi23!ion<‘Ad<fcess: 

§> f Cci/^^c 


Teleohofw: ^ j 


rt-fiS 33 


£-Ma>t Address: 


>h!ok. 



IS © so ccPo 



ERIC 



ICdO Holfo voay Ctu^ nuc 

fra^osco^ cA 9 V 7 - 5 J-. 



iii. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 



If permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source, 
please provide the following information regarding the availability of the document. {ERIC will not announce a document unless it is 
publicly available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are 
significantly more stringent for documents that cannot be made available through EDRS.) 




IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 



If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and 
address: 




V. WHERE TO SEND THIS FORM : 



Send this form to the following ERIC Clearinghouse: 



However, If solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being 
contributed) to: 



ERiC Processing and Reference Facility 
4483-A Forbes Boulevard 
Lanham, Maryland 20706 



Telephone: 
Toll Free: 
FAX: 
e-mail: 
WWW: 



301-552*4200 

800-799-3742 

301-552-4700 

ericfac@inet.ed.gov 

http:/feHcfacllity.org 



EFF-088 (Rev. 2/2001) 




